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Examining Effects of PreACT Adoption on 
College Readiness Outcomes 


Jeff Allen, PhD 


Based on the first cohort of students who took the PreACT® in 10th grade and the ACT® test in 11th 
grade, this study examines the effects of PreACT adoption on ACT test scores, participation in 
challenging high school courses, interest-major fit, and college score sending behavior. Adoption of the 
PreACT led to an increase in ACT Composite score of 0.23 score points, which is comparable to one 
month of instruction. Adoption of the PreACT also led to small increases in interest-major fit and out-of- 
state score sending. 


Introduction 


The PreACT test is a multiple choice test that provides students with practice for the ACT test and measures of 
progress towards college readiness. The PreACT can be administered to students in any grade but is most commonly 
administered to 10th graders. This paper documents a study that examined the effects of schoolwide PreACT adoption 
on college readiness outcomes for the first cohort of students to have taken the PreACT and ACT tests. The outcomes 
include ACT test scores, participation in challenging high school courses, interest-major fit, and college score sending 
behavior. Table 1 provides rationale for why administration of the PreACT could affect these outcomes. 


boo) | (-Ye C= aX-¥-(o ll al-eoxomOLUi Coxe) tal Rationale for PreACT Effect 


By taking the PreACT, students are exposed to content and test items that 
simulate the ACT testing experience, leading to greater familiarity of the 

ACT test scores knowledge and skills tested by the ACT. Further, by receiving feedback on their 
PreACT performance, students may focus on areas that need more work and may 
be more likely to engage in other preparation and learning activities. 


Participation in advanced high Through the feedback they receive from the PreACT test, students learn that they 
school coursework and upper- are ready for advanced high school coursework. Further, if PreACT feedback 

; stimulates college planning, students may be more likely to take challenging high 
level math and science courses school courses to improve their readiness for college. 


Students who take the PreACT test may take the Interest Inventory and engage in 
Int t arti college major exploration. Increased exploration may lead to greater fit between 
EtSrSslemalar tl their personal interests and the major they plan on entering when they take the 
ACT test in 11th grade. 


Students who take the PreACT test may engage in more college exploration and 
planning, which could result in them sending ACT test scores to more colleges 

College score sending behavior and to more out-of-state colleges. Moreover, they may participate in the PreACT 
Educational Opportunity Service (EOS), which could increase their exposure to, 
and knowledge of, more colleges. 
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Study Design: The Difference-in-Difference Approach 


The effects of PreACT adoption are examined using a difference-in-difference (DiD) approach (Figure 1). PreACT 
schools administered the ACT test to all 11th graders in spring 2017 and spring 2018 and the PreACT test to 10th 
grade students in 2016-2017. Therefore, most students who were in the spring 2018 11th-grade cohort took the 
PreACT test, while none of the students who were in the spring 2017 11th-grade cohort took the PreACT test. Figure 
1 shows a hypothetical scenario where the mean ACT Composite score increased from 19.25 in 2017 to 19.50 in 
2018 for PreACT schools. 


Non-PreACT schools also administered the ACT test to all 11th graders in spring 2017 and spring 2018 but did not 
administer the PreACT test for either cohort. Figure 1 shows a hypothetical scenario where the mean ACT Composite 
score increased from 19.0 in 2017 to 19.05 in 2018 for non-PreACT schools. The first difference measures 
improvement in college readiness outcomes from the first cohort (2017) to the second cohort (2018) and is calculated 
for both PreACT and non-PreACT schools. In Figure 1, PreACT schools improved by 0.25 score points and non- 
PreACT schools improved by 0.05 score points. The DiD is calculated as the difference in improvement for PreACT 
schools versus non-PreACT schools (e.g., 0.25-0.05=0.20) and estimates the effect of PreACT adoption. 


Figure 1. Hypothetical Difference-in-Difference for Measuring Effects of PreACT Adoption 
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Sample and Data 


To be included in the analysis, schools must have administered the ACT test to 11th graders through a state or 
district program in spring 2017 and spring 2018 and the number of students tested in the two years must have been 
relatively stable.’ ACT test records from spring 2018 were matched to PreACT test records from 2016-2017. Schools 
that had at least 75% of student records matching to a PreACT test record were considered PreACT schools.” 
Schools that had less than 2% of student records matching to a PreACT test record were considered non-PreACT 
schools. Schools from states that had some PreACT schools and some non-PreACT schools were included so that 
effects of PreACT adoption can be distinguished from other changes in the states. Schools from North Carolina and 
Oklahoma were not included in the analysis because virtually all schools were PreACT schools and so effects of 
PreACT adoption cannot be distinguished from other changes in those states. Table 2 summarizes the number of 
schools and students included in the analysis, as well as their background variables. 


There were 400 PreACT schools and 3,972 non-PreACT schools from 31 states. The number of students who took 
the ACT test was very similar across the two cohorts. For both PreACT schools and non-PreACT schools, the 
demographic breakdowns were very similar across the two cohorts. Relative to PreACT schools, non-PreACT 
schools had larger concentrations of African American (13% vs. 8%), Hispanic (12% or 13% vs. 9%), missing 
race/ethnicity (14% vs. 9% or 10%), and missing parent education level (40% and 35% vs. 32% and 24%) (Table 2). 
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Table 2. Schools and Students Included in Analysis of Effects of PreACT Adoption 


PreACT PreACT Non-PreACT Non-PreACT 
NWETarlel(a Yel plete) Ye pele) schools schools 

2017 2018 2017 2018 
PT ON accor lactated ee iat ikl Ls 
% who took PreACT 0% 83% 0% <1% 


Male 49% 49% 48% 48% 


African American 8% 8% 13% 13% 


Other 4% 5% 5% 5% 


Parent education level 


Bachelor’s degree 25% 22% 18% 16% 


Statistical Analysis 


College readiness outcomes were compared using the DiD approach. The outcomes included: 


e ACT test scores (English, mathematics, reading, science, and Composite) 


e Number of subjects (out of five) for which a student took advanced placement, accelerated, or honors 
courses 


e Taken (or plan to take) calculus, other advanced math courses, and physics 


e __Interest-major correlation (the correlation of vocational interests with the environment of student’s planned 
college major) 


e Sending ACT scores to at least four colleges* 


e Sending ACT scores to at least one out-of-state college* 


For each school and each cohort, the mean outcome was calculated. The cohort difference (mean for 2018 — mean 
for 2017) was then calculated for each school. Next, the mean cohort difference was calculated for PreACT schools 
as the mean of the school mean differences. Similarly, the mean cohort difference was calculated for non-PreACT 
schools as the mean of the school mean differences. The DiD was calculated as the difference between the mean 
cohort difference for PreACT and non-PreACT schools. 


PreACT and non-PreACT schools could have underlying differences that impact the DiD. A propensity score 
weighting approach (Austin, 2011) was used to ensure that the PreACT and non-PreACT schools were similar on 
several covariates, including the ratio of students tested in 2017 and 2018 (N2o18/N2017), mean outcome at baseline 
(e.g., for the 2017 cohort), changes in demographics from 2017 to 2018 (proportion male, African American, 
Hispanic, parents with no college degree), and changes in ACT month tested. A logistic regression model was used 
to predict group membership (PreACT or non-PreACT) using the covariates. The logistic regression model produces 
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a predicted probability of being a PreACT school, and this predicted probability is known as the propensity score (ps). 
After assigning inverse probability of treatment weights to PreACT schools (weight = 1/ps) and non-PreACT schools 
(weight = 1/(1-ps)), the two groups are balanced on the covariates. A weighted regression model was used to 
estimate the adjusted DiD and determine if it was significantly different than zero (e.g., if the mean cohort difference 
for PreACT schools was different than the mean cohort difference for non-PreACT schools). 


Some effects of PreACT adoption are contingent on student participation in certain components of the PreACT. For 
example, through the Educational Opportunity Service (EOS), students may be provided information about 
prospective colleges that they may not have received otherwise. To realize the benefits of EOS through the PreACT, 
students must opt-in when they take the PreACT test so that their name and other information is provided to colleges. 
Similarly, to realize the full benefit of PreACT’s college major and career exploration tools, students must take the 
PreACT Interest Inventory. In this study, among students in the PreACT schools who took the PreACT, 68% opted in 
to the EOS and 90% took the Interest Inventory. In addition to the DiD analyses, we examined outcomes within 
PreACT schools, comparing students who did and did not participate in the Interest Inventory and EOS. The 
propensity score weighting approach (Austin, 2011) was used to ensure that the participants and nonparticipants are 
similar on PreACT Composite score, gender, race/ethnicity, and parent education level. 


Comparison of College Readiness Outcomes 


Table 3 summarizes the results of the DiD analyses. The results suggest a small positive effect of PreACT adoption 
on ACT test scores. PreACT schools improved 0.378 English score points more than their non-PreACT counterparts. 
The effects for mathematics (0.168), reading (0.199), science (0.258), and Composite (0.231) were smaller. A 
previous ACT study estimated the average gain in ACT scores per month of instruction (Camara & Allen, 2017). 
Using those results, the estimates of the PreACT effects from the DiD model can be compared to the months of 
instruction typically needed to obtain the same gain (Table 4). For example, mean ACT Composite score typically 
increases by 0.227 points per month of instruction. The PreACT effect on Composite score (0.231) is therefore 
comparable to 1.02 (0.231/0.227) months of instruction. 


Table 3. Comparison of College Readiness Outcomes 


Neva Non- Neva 
Bren a ene PtenGl  ereacrt | (Preach =) (Preach Adjusted 
Outcome Yeiatele) is schools schools Schools schools sehools ae 
2017 Zoe 2017 2018 Diff. 
ACT English 19.589 19.574 -0.015 18.032 17.785 -0.247 0.232 **0.378 
ACT Reading 20.521 20.359 -0.163 19.204 18.961 -0.243 0.080 **0.199 
i ACTScience ———~=~—«é02« 120.223 0.091 Ss 19.091 18.848 = 0.242 0.151 0.258 


Four scores sent 0.541 0.535 -0.006 0.480 0.491 0.011 -0.018 0.008 


DiD = difference in difference, ** p-value < 0.01, * p-value < 0.05 


The DiD estimates for the other outcomes were generally small. For example, the percentage of students sending at 
least one score to an out-of-state college increased slightly for PreACT schools from 44.5% to 45.1% and decreased 
slightly for non-PreACT schools from 42.3% to 42.0%. The DiD was 0.01 (1.0%), and the adjusted DiD was 0.018 
(1.8%) and statistically significant. The adjusted DiD was also positive and significant for interest-major correlation, 
suggesting a small effect of PreACT adoption on interest-maijor fit. 
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Table 4. Estimated Effects of PreACT Adoption on ACT Scores 


PreACT effect PreACT effect 


ACT gain per month 


Subject area Ceres adjusted months of 
DiD estimate instruction 
English 0.277 0.378 1.36 
“Mathematics = tt—s—s 0199 0168 O84 


Science 


DiD = difference in difference 


Additional analyses were conducted to examine differences in outcomes for students who did and did not participate 
in the EOS and the Interest Inventory components of the PreACT. This analysis was confined to students in PreACT 
schools who took the PreACT. ACT score sending behavior was compared for EOS participants and nonparticipants. 
Interest-major correlation was compared for Interest Inventory participants and nonparticipants. Because the 
participants and nonparticipants could have underlying differences related to the outcome, a propensity score 
weighting (PSW) approach was used to balance the two groups.® By weighting the sample, an adjusted difference 
between participants and nonparticipants can be calculated. 


Students who completed the PreACT Interest Inventory had higher correlation between their ACT planned major and 
their vocational interests (0.381 for participants versus 0.359 for nonparticipants, difference=0.022). After applying the 
PSW and linear regression, the adjusted difference (0.035) was larger than the unadjusted difference and was 
statistically significant (Table 5). Students who opted into EOS for the PreACT were more likely to send their ACT 
scores to at least four colleges (0.567 for participants versus 0.447 for nonparticipants, difference=0.120). After 
applying the PSW and logistic regression, the adjusted difference was 0.029 and was statistically significant. 
Students who opted into EOS for the PreACT were also more likely to send their ACT scores to at least one out-of- 
state college (adjusted difference=0.019). 


Table 5. Comparing Outcomes Among PreACT Component Participants and Nonparticipants 


Participation Mean Outcome Mean Outcome Adjusted 
Outcome : oe Difference ; 

N7Tarele\(= Participants Nfoyavor=tatletlor= Tales difference 
Int t-maj Int t Inv. 
ee eur 0.381 0.359 0.022 **0.035 
correlation completion eee 
Four scores sent EOS opt-in : 0.447 0.120 **0.029 


**p-value<0.01 


Conclusion 


This study provides evidence that schoolwide adoption of the PreACT leads to small improvements in ACT test 
scores. This effect could be due to exposure to test content and items that mimic the ACT test, or by the feedback 
students receive from taking the PreACT. The study also provides evidence of small effects of PreACT adoption on 
interest-major correlation and out-of-state college score sending. Further, within PreACT schools, students who 
participated in the PreACT EOS were more likely to send their ACT scores to at least four colleges and at least one 
out-of-state college. And, students who completed the PreACT Interest Inventory had slightly higher fit between their 
interests and planned major when they took the ACT. The study does not provide evidence of PreACT effects on 
outcomes related to taking challenging high school courses. 


There are several limitations of the study that should be understood. First, it’s possible that the PreACT test was not 


the only new intervention introduced between the two study cohorts. It’s possible that both PreACT and non-PreACT 
schools introduced other interventions that could affect the study outcomes or confound the relationship of PreACT 


Copyright ® 2018 by ACT, Inc. All rights reserved. | R1724 


Oo 


ACT Research & Policy | Technical Brief | October 2018 


adoption and study outcomes. Second, there are no data available on how PreACT results were used within schools. 
If the PreACT test was taken but no action was taken to use the results, we would not necessarily expect PreACT 
adoption to lead to improved outcomes. There is likely significant variation across schools in how PreACT results are 
used, and this study does not provide insight into best practices for using PreACT results. Third, the study only 
measured the impact of PreACT adoption for the first cohort of students tested (e.g., 11th graders of spring 2018). It’s 
possible that PreACT will have a larger impact as educators have more time to learn to use the results and integrate 
the assessment with other programs designed to improve college readiness outcomes. 
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Notes 
1. Neo17 and Nzo1g must have been within 50% of one another. 
2. Some schools had a very small percentage of students with a PreACT test record, likely due to student migration from a 


PreACT school. 


3. Students may send ACT scores to four colleges at no additional charge. 

4. Among four scores sent at no additional charge. 

5. Inverse probability of treatment weights were assigned (Austin, 2011) based on propensity scores for participation in 
the PreACT component. The propensity score models estimated the probability of participation, based on PreACT 
Composite score, student gender, race/ethnicity, and parent education level. 
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